Bartowski writes about a new method for optimizing GGUF quantizations by using per-tensor layout maps. By analyzing the sensitivity of different tensor types through extensive experimentation, the author developed a solver that allocates more bits to sensitive components (like embeddings and specific attention projections) while "crushing" less critical tensors. This approach aims to maximize performance per bit and restores semantic meaning to quantization naming conventions by ensuring tiered variants like _S, _M, and _L contain consistent proportions of high-bit tensors.
- Token embeddings are highly sensitive and benefit significantly from higher precision.
- Sensitivity follows a "U" shape relative to model depth (higher at the beginning and end).
- Small attention projections such as `attn_v` and `ffn_up` are among the most sensitive tensor types per bit.
- A "canary" test system is used to verify if new layout maps generalize well across different model architectures before deployment.
- Token embeddings are highly sensitive and benefit significantly from higher precision.
- Sensitivity follows a "U" shape relative to model depth (higher at the beginning and end).
- Small attention projections such as `attn_v` and `ffn_up` are among the most sensitive tensor types per bit.
- A "canary" test system is used to verify if new layout maps generalize well across different model architectures before deployment.
